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Abstract — Suppose that we are given two sources 5*1, ^2 and an 
"error-control" family Q. We assume that we lossy-code 5*1 with 
Q-acceptable alphabet Vi and S2 with Q-acceptable alphabet V2- 
Consider a new source S which sends a signal produced by source 
Si with probability ai and by source S2 with probability 02 — 
1 ai. We provide a simple greedy algorithm which constructs 
a Q-acceptable coding alphabet P of S such that the entropy 
h{P) satisfies: 

h{r) < ai h{Pi) + 02 h{P2) + 1. 

In the proof of the above formula the basic role is played by a 
new equivalent definition of entropy based on measures instead 
of partitions. 

As a consequence we obtain an estimation of the entropy 
and Renyi entropy dimension of the convex combination of 
measures. In particular if probability measures pi,/x2 have 
entropy dimension then 

dim£;(ai/ii + 02/12) — aidimB(/ii) + a2dimE{n2)- 

In the case of probability measures in this allows to link the 
upper local dimension at point with the upper entropy dimension 
of a measure by an improved version of Young estimation: 

dimB(/i) < / D^{x)dii(x), 

where D^{x) stands for upper local dimension of ^ at point x. 

Index Terms — Entropy coding, entropy dimension, lossy cod- 
ing, mixture of sources, Renyi information dimension, Shannon 
entropy. 



L Introduction 

THE classical entropy introduced by C. E. Shannon [1] 
and the entropy dimension' defined by A. Renyi [2] 
play a crucial role in information theory, coding, study of 
statistical and physical systems [3]-[6]. In information theory, 
the entropy is understood as an absolute limit of the best 
possible lossless compression of any communication. The 
entropy dimension in turn can be interpreted as a rate of 
convergence of the minimal amount of information needed 
to encode randomly chosen element with respect to maximal 
error decreasing to zero. 



A. Motivation 

To explain our results, let us first recall that given a 
probability measure /i on a space X and a countable partition 

The authors are with Institute of Computer Science, Jagiellonian University, 
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'it is sometimes called Renyi information dimension. 



V of X into measurable sets, we define the entropy of fi with 
respect to V by the formula 



(1) 
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where sh{x) := — xlogjo;. As we know the entropy cor- 
responds to the statistical amount of information given by 
optimal lossy-coding of X by elements of partition V, where 
V plays the role of the coding alphabet. Motivated by the idea 
of Renyi realized by the entropy dimension, we generalise the 
above formula for arbitrary measurable cover Q of X by 



Hi^^■, Q) := 

mf{h{ii;V) 



V is a partition of X and V ^ Q}. 



(2) 



The family Q is interpreted as a maximal error we are allowed 
to make in the lossy-coding. We accept only such coding 
alphabets V, in which every element of P is a subset of a 
certain element of Q (if this is the case we say that V is 
Q-acceptable). 

Remark I.l. The simplest natural case of such error-control 
family Q for classical random variables is given by the set Bg 
of all intervals in M with length S. Then to find H{fJ,;Bs) we 
need to consider the infimum of entropies of all lossy-codings 
h{p', V), where the elements of V have length not greater than 
S. 

A. Renyi considered the above error-control family Bs in his 
definition of entropy dimension [2] (he also studied the more 
general case of metric spaces when Bs denoted the family of 
all balls with radius S). One can also encounter in the general 
metric spaces the family of sets with diameter 5 or in the case 
of of cubes with edge-length 5. 

Our basic motivation in the paper was the following prob- 
lem: 

Problem I.l. Suppose that we are given an error-control 
family Q and two sources 5*1, S2 in X (represented by 
probability measures /ii,/i2 on X). Let us consider a new 
source S which sends a signal produced by source 5*1 with 
probability ai and by source ^2 with probability 02 = 1 — oi. 
Source 5 is a mixture of Si and 52. The question is what is 
the entropy of source S with respect to the error Q? 

In other words we are interested in estimation of H{aiiii + 
a2/i2; Q) in terms of H{^il■, Q) and H{p2; Q)- 

Observation I.l. Observe that if elements of Q are pairwise 
disjoint then the answer to the above problem is trivial as by 
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the subadditivity of the function sh we have 

H{ii; Q) = Q) = ^ sh{n{Q)) (3) 

= sh{aiiii{Q) + a2/X2(Q)) (4) 

< sh{aiiJ.i{Q)) + sh{a2lJL2{Q)) (5) 

= ai Q) + aa if(/X2; Q) + sh{ai) + 5/1(02). (6) 

7b see that the above estimation is sharp it is sufficient to 
consider a source Si which sends only signal and source 
S2 which sends signal 1. Clearly, H{Si) = H{S2) ~ 0. Then 
the entropy of the source S which sends signal generated by 
Si with probability ai and S2 with probability 02 is exactly 
ai HiSi) + 02 H{S2) + sh{ai) + 5/1(02). 

B. Main Results 

In our main result, Theorem III.l, we show that the formula 
calculated in the above observation: 

F(ai/ii + 02/12; Q) 

< 01 i7(/ii; Q) + 02 i7(/i2; Q) + sh{ai) + 5/1(02) 

is valid in the general case, that is when Q is an arbitrary 
measurable cover of X. The proof of our main result reUes 
on a new definition of entropy based on measures instead of 
partitions, which we call weighted entropy. We provide an 
algorithm, which for given alphabets Vi,V2 and measures 
^1, ^2 allows to construct "joint" alphabet V satisfying above 
inequality. 

Remark 1.2. We would like to add here that our idea of 
weighted entropy is indebted to the notion of weighted Haus- 
dorff measures considered by J. Howroyd [7], [8]. The advan- 
tage of weighted Hausdorff measures over the classical ones 
is well-sunmiarised by words of K. Falconer [9, Introduction]: 
"Recently, a completely different approach was introduced by 
Howroyd using weighted Hausdorff measures to enable the 
use of powerful techniques from functional analysis, such as 
the Hahn-Banach and Krein-Milman theorems." Making use 
of weighted Hausdorff measures Howroyd proves that 

dim H{X)+dimH [Y) < dim h{X xY), (8) 

where dimi/(X) is the Hausdorff-Besicovitch dimension of 
X. 

For the precise definition of weighted entropy we refer the 
reader to the next section. We would only like to mention that, 
roughly speaking, weighted entropy provides the computation 
and interpretation of the entropy with respect to "formal" 
convex combination aiVi +02^2, where Vi,V2 are partitions 
(which clearly does not make sense in the classical approach). 
This operation is crucial in the proof of formula (7), whereas 
the second important part is played by Theorem 11.1, which 
proves that the weighted entropy is equal to the classical one. 

As an easy consequence of (7) in Theorem IV. 1 we obtain an 
estimation of the entropy dimension of the convex combination 



of measures. This result can be summarised as follows (see 
Corollary IV. 1): 

Let fii and fi2 be probability measures which have entropy 
dimension and let 01,02 G (0, 1) be such that 01 + 02 = 1. 
Then oi/ii + 02/^2 has entropy dimension and 

dim£;(oi^i + 02^2) = aiA\mE{ni) + a2dim£;(Ai2), (9) 

where dim£;(-) stands for the entropy dimension of a given 
measure. 

In the case of measures in this allows to combine the 
local upper dimension i5^(-) with the upper entropy dimension 
dim£;( ) and improve Young estimation of the upper entropy 
dimension [10]: 

dta^B(/i) < / Du,{x)dn{x). (10) 

II. Weighted Entropy 

From now on, if not stated otherwise, we assume that 
{X, E, p) is a probabihty space. The set of probability mea- 
sures on (X, E) will be denoted by Mi(X, E). When we 
consider a set of all measures then we will write M{X, E). 

A. Shannon Entropy and Deterministic Coding 

We begin with the definition of /i-partitions, which will play 
a role of a coding alphabet. 

Definition II.l. Let C E. We say that P is a ji-partition 
(of X) if V is countable family of disjoint sets and 

fi{X\[jP)=0. (11) 
Per 

Consequently every element x E X, which can be randomly 
drawn (except for possibly elements of measure zero), is coded 
deterministically by the unique P gV such that x G P. 

Then the entropy [1] of /x-partition is defined as follows: 

Definition II.2. Let P c E be a /^-partition of X. We define 

jjL-entropy of V by 

/i(/.;P):= ^5/.(MP)), (12) 

where sh : [0, 1] — >■ IR+ is the Shannon function, i.e. 

._ / -^•log2(a;) fora;e(0,l], 
sii{x) .- I g for a; = 0. ^ ^' 

Let us mention that sh is a continuous, concave and 
subadditive function. 

Classical /z-entropy is defined with use of disjoint sets, 
which is a very restrictive condition. It implies that we have 
fixed one alphabet V in our lossy-coding. However, this 
alphabet does not have to be optimal. In other words, there 
may exists another Q-acceptable alphabet V' , which provides 
less entropy than V (we assume that V is also Q-acceptable). 
Thus it would be better to make a coding with use of V 
rather than with V. Therefore we will generahse the entropy 
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for any error-control family. The error-control family can be 
an arbitrary family of measurable subsets of X. 

We say that family V is finer than Q (which we write V -< 
Q) if for every P e V there exists Q € Q such that P C Q. 
When V is interpreted as a coding alphabet we may simply 
say that V is Q-acceptable. 

Definition 11.3. Let Q C S. We define the ^-entropy of Q by 
Hili; Q) := 

m^{h{^.■,'P) G [0,00] : P is a /i-partition and P -< Q}. 

(14) 

Observe that if there is no /x-partition finer than Q then 
directly from the definition^ -ff(M) 2) = 00. Moreover, if Q 
itself is a /i-partition of X then trivially-' H{^^\ Q) = h{fJ.; Q). 
This observation imphes that /x-entropy of Q is defined 
properly for /x-partitions as well as for families of measurable 
subsets of X. 

B. Weighted Entropy and Random Coding 

Motivation of the weighted entropy is the following obser- 
vation. Given error-control family Q in the classical approach 
we consider only Q-acceptable deterministic codings V. More 
precisely we always code a point a; e X by the unique P^ €V 
such that X G Px- 

However, if we do not insist on being deterministic in our 
coding, we could alternatively encode point x by another set 
P' G S such that x G P' and for which there exists Q' E Q : 
P' C Q' ■ In this subsection we formalise this idea, namely we 
do not fix a Q-acceptable alphabet V but we allow any random 
coding demanding only that x can be encoded by Q G Q iff 
a; G Q. Such a random coding might theoretically give lower 
entropy than the original one. 

We make it precise in the following way. We define the 
space of functions from a family of measurable subsets of X 
into a set of measures on X: 

Wifi; Q) {m : Q 9 Q mg G M{X, S) : 

mQ{X \Q) =0 for every Q G Q and J2QeQ = 

(15) 

Thus given m G W{fi; Q) and Q G Q, the value of mQ{X) 
denotes the probabihty that an arbitrary point x € X is coded 
by Q (and in that case x £ Q with probability one). Observe 
also that every function m G W{fi; Q) is non-zero on at most 
countable sets of Q. 

Finally we define weighted /x-entropy of a given m G 
W(/x; Q): 

Definition II.4. Let Q c S. We define the weighted fi-entwpy 
of m G W{ii; Q) by 

/i,w(At;m) := ^ sh{mQ{X)). (16) 
QeQ 

The weighted /x-entropy of Q is 

HwilJ-; Q) := inf{/iH.(/x;m) G [0,oo] : m G W{fi, Q)}. 

(17) 

^We put inf(0) = 00. 

'We can consider another //-partition P ^ Q of X but due to subadditivity 
of sh we get h{^l■, Q) < h{n; V). 



The following remark explains the importance of the for- 
mulation of weighted entropy. 

Remark II.l. Given functions mi,m2 G W{n\ Q) and num- 
bers 0,1,02 G [0, 1] such that ai -|- 02 = 1 we are allowed 
to perform convex combinations aitrii + a2m2 in the space 
W{ii\ Q). Therefore we can compute the weighted /x-entropy 
of a combination /in/(^;aimi + a2m2) while the symbol 
hip-] cLiPi -h 02^2) does not make sense for /x-partitions 
■Pi, 7^2- This property wiU help us to find an estimation of en- 
tropy of convex combination of measures ff(ai/(Xi + 02/1x2; Q) 
for Q c S. 

C. Classical Entropy Equals Weighted 

In this section we show that the classical /x-entropy of a 
family of measurable sets Q equals to the weighted /x-entropy 
of Q, i.e. 

Hw{li-:Q) = H{ii:Q). (18) 

It seems natural that every deterministic coding is a particular 
case of a random one. We will show it in the following 
proposition. 

Let us denote the restriction of measure /x to A G S by 

/X|^(B) :=/x(AnS) (19) 

for every _B G S. 

Proposition II.l. Random way of coding allows possibly more 
freedom than the deterministic one, i.e. 

Hw{f.Q)<H{ii;Q) (20) 

for every family Q C S. 

Proof: Let us first observe that if there is no /t-partition 
finer than Q then ij(/t; Q) = 00 and the inequaUty holds 
trivially. 

Thus let P be a /t-partition finer than Q. As P Q, for 
every P G P there exists Q G Q such that P C Q. Hence 
we obtain a mapping it : V ^ Q satisfying P C 7r(P). We 
define the family 

:= {Pglgee, (21) 
where Pq ■= U P. Let us notice that Pg is a /i- 

P:7r{P)=Q 

partition and P ^ Vq -< Q. Finally, we put m : Q 9 Q 
/X|p^ gM(X,S). 

Since Pq is a /x-partition and Pq C Q for every Q £ Q 
then 

xriQiX) = J2 M|P«(Q) = E ^'(PQ^ = '"(^)- (22) 

QeQ QeQ QeQ 

Moreover, for every Q G Q 

mQ{X \ Q) - m|Pq \ Q) < 1^\q{X \ Q) = 0. (23) 

Thus m G H^(/t; Q). Making use of subadditivity of sh we 
obtain 

/ivy(/x;m) = MmqiX)) = ^ s/i(/x|Pq (X)) (24) 
QeQ QeQ 
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= ^Hf^iPQ)) = E «^('"( U ^)) (25) 

QeQ QeQ P:tt(p)=q 

QeSP:7r(P)=Q PG-P 

(26) 

We conclude that Hw{n] Q) < Hin; Q). ■ 
The opposite inequality is more difficult to prove. To do this 
we will need an additional proposition. Given m G W{^; Q) 
we wiU construct a /U-partition P finer than Q with not greater 
entropy. 

Proposition n.2. Let Q = {Qi}i£i be a family of measurable 
subsets of X, where either I = N or I = {!,..., A''} for a 
certain G N. Let m G W(/Lt; Q). We assume that 

. ^l{x\y] Qi) = 0, 

iei 

• the sequence I 3 i ^ m.Q^{X) is nonincreasing. 
We define the family V = {Pjjjgj cT, by the formula 

i-l 

Pi := Qi, Pi := Oi \ IJ Qkfor iel,i>2. (27) 
fe=i 

Then V is a ^-partition, V -< Q and 

hw{li\m)>h{n\V). (28) 

Proof: Let us observe that by the definition of V, we 

have V <Q. Moreover, since n{X \ U Qj) = and \J Pi = 

iei iei 
U Qi, we get that P is a /^-partition. 

iGl 

To prove (28) we define sequences {xi)iQi C [0,1] and 
{yi)iei C [0, 1] by the formulas 

xr-=mQ^{X)=mQ,{Qi), (29) 

Vi := fi{Pi) (30) 

for i G I. Then 

E^. = M^) = Ey- (31) 

iei iei 

Directly from the assumption we conclude that (a;,),^/ is a 
nonincreasing sequence. Moreover, for every n € I: 

n n n 

Ea;i = E"^0*('3') = (E"^*)(<3iU...UQ„) (32) 

i=l i=l i=l 

n n 

<M(QiU...UQ„) = EM^i)=Ey- (33) 

i=l i=l 

We have obtained that 

n n 

Yxi<Yyiiorn&I. (34) 

By applying the version of Hardy-Polya-Littlewood Theorem 
(see Appendix A for details) for sequences {xi)i^i, {yi)iei 
and the concave function sh we conclude that 

hwilJ-'^m) = Ysh{mQ,{X)) = Ysh{xi) (35) 
iei iei 

>^s%,)=^s/i(/x(P,)) = /i(m;^)- (36) 
iei iei 



As a direct corollary we obtain that both random and 
deterministic coding provide the same entropy. 

Theorem II.l. Let Q C S. Then weighted entropy coincides 
with the classical entropy, i.e. 

Hwip:Q) = H(p:Q). (37) 

Proof: Clearly by Proposition II.l, we get Hw{l-i; Q) < 
H{ii; Q). 

To obtain the opposite inequality, let us first observe that if 
W{ijl; Q) = then Hw{^J^■, Q) = oo and trivially (a*; Q) > 
Hii^; Q). 

We discuss the case when W{^-, Q) ^ 0. Let m G W{fj,; Q) 
be arbitrary. We define the family of measurable subsets of X 
by 

Q := {Q e Q : mQ{X) > 0}. (38) 

Let us notice that Q is a countable family since ^ mq {X) = 

QeQ 

1. Clearly, m := m^Q G W{ii;Q). Moreover, Q ^ Q and 
/iH^(/i;m) = hw{lJ';m). 

As Q is countable, we may find a set of indices I C N 
such that Q = {Qi}iei and the sequence I B i ^ mg. (X) is 
nonincreasing. Making use of Proposition II.2 we construct a 
/U-partition V -< Q, which satisfies 

hw{f^;m) > h{n;V) (39) 

This completes the proof since V -< Q -< Q and /iv^(/i;m) = 
hwin;m)>h{fJ.;V). ■ 
As we proved the equality between classical and weighted 
entropy, we wiU use one notation Q) to denote both 
classical and weighted /x-entropy of Q C S. 

III. Entropy of the Mixture of Sources 

A. Estimation of the Entropy 

We return to Problem I.I. We are given two sources 81,82, 
which are represented by probability measures ^1, ^2 respec- 
tively. Suppose that we have fixed error-control family Q C S, 
which defines the precision in the lossy-coding elements of X. 
Let us consider a new source 8 which sends a signal produced 
by 81 with probability ai and produced by 6*2 with probability 

02. We are interested in estimation of the entropy of 8 (mixture 
of Si and ^2) with respect to Q in terms of Q) and 
H{fJ.2', Q)- In other words we would like to measure how 
much memory we need to reserve for information from source 
8 providing that we know the mean amount of information 
needed to encode elements sent by 81 and 5*2 separately. 

We will consider a general case: we assume n € N sources 
81, . . . ,8n. Let us begin with a proposition. 

Proposition in.l. Let n G N and let at G (0,1) for 

n 

k G {l,...,n} be such that ^ Ofe = 1. Let {nk}k=i C 

k=l 

n 

Mi{X, S). We put At := E "feMfc € Mi{X, S). 
fe=i 
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IfV is a ii-partition of X then V is a fj,k-partition of X We denote ^ := X] Ofe/Xfe. Let £ > be arbitrary. By the 

for k G {1, . . . ,n} and , ^ .. r ^ , • • -r- ^ , ^ , 

denmtion oi entropy, we find a /x-partition V finer than Q such 

n that 
h{l^;r)>y2akh{nk;'P)- (40) 

fc=i H{n;Q)>h{fi;V)-s. (50) 

// Q C S anc/ m'' £ VK(Aifc; Q) /o?" € {1, . . . ,n} ?/ien 



m := X] «/c"^'^ € W(/Lt; Q) and 
fe=i 



Then by Proposition III.l, we have 



" « h{l^;P) = hC^akiik;V) (51) 

hw{l^;m) <2_^akhw{lJ'k]m ) + 2_^sh{ak). (41) fe=i 



fe=i fe=i 



Proof: Clearly, T' is a /ife -partition for every fc £ " " 

{1, . . . , n}. As a direct consequence of the concavity of the 

Shannon function we obtain that ^==1 

n 

Mm; P) = E ^MM^)) = E «'^^^(^)) (42) Consequently by (50), 

Per Pev fe=i „ 

^ " " H{i^;Q)>h{f^;V)-e>y2akH{fik;Q)-e. (53) 

> E E"'^*^^'^'^^^)) = E«fe^(^fc;^) (43) k=i 

Per fe=i fe=i 

which proves (40) prove the second inequahty. Again by the definition, for 

It is easy verify that m e W{ii; Q). To prove (41) we use ^"^^ A; e {1, ... n} we find m'^ € W{^lk] Q) such that 
subadditivity of the Shannon function and property: sh{ax) = 



e 



ash{x)+xsh{a). hw [tik] ra") < H{Hk\Q) + -■ (54) 



n 



hwifJ-; m) = E ^'^(E (44) Then by Proposition IH.l and (54), we obtain 

QeQ fe=i 

n 

< 2^ 2^s/j(ofcmQ(X)) (45) 
QeQfe=i 

" n 

= E Et'^fc*''("^Q(^))+^'^('^fc)"^Q(^)] (46) <E["fc'^M'('"fc;"^'')+«'^("'^)] (56) 

fe=i QeQ 

n n 

= E«fe^w'(A<fe;"^'') + E'^'**^'^'^^- (4"^) 

fe=i fe=i <Y^[akH{nk;Q) + sh{ak)]+e, (57) 

■ 

Making use of Proposition 111.1 we can estimate the entropy 
of convex combination of measures, which is the main result "^^""^ completes the proof as £ > was an arbitrary number, 
of the paper: „ * 

™ TTT1 r , ^ ji, ^ rn iw Clearly, s^K) < log2(n). Thus the assertion (49) of 

Theorem III.l. Let n e N and let ak G [0, IJ for k=i 



k € be such that ^ = 1. Let {fik}k=i ^ 



Theorem in.l can be also rewritten as 



fe=i 



M,{X, S). // Q CE then ^ ^ „^ ^ j^^^^^^^ (53) 



fe=i fc=i 



//(E"fc'"fe;2) > E"fc^('"fc;2) (4^) 

fe=i fe=i When we consider a combination of two probability measures 

dfid then we get directly: 

/A n\^\^ u( n\-L\^ h( \ fAQ\ Corollary 111.1. Let ai,a2 e (0,1) be such that ai+a2 = I. 

H (2_, akHk, ^ 2^ak HKi^k, + 2^ sniak). ^.^^^ probability measures /xi, /xa an^/ a/awj'/y of measurable 

~ ~ ~ subsets Q of X, we have 

Proof: We consider the case when all considered en- 

tropics are finite because if Hijik^Q) = oo for a certain ^^^^^^ ^ > ^(^^. q) + ^(^3; Q), (59) 

K € {l,...,n| then also if (/u; Q) = cx) and the proof 

is completed. Moreover, without loss of generality, we may 

assume that afe ^ for every fc e {1, . . . , n}. F(aiMi +012^2; Q) < ai Q) -ha2 i?(/X2; Q) -h 1. (60) 
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B. Practical Algorithm for Finding "Joint" Coding Alphabet 
of the Mixture of Sources 

A practical question is how to construct Q-acceptable 
coding alphabet V form given alphabets Vi and V2 such that 

h{aiHi + a2lJ.2;'P) < ^^^^ 
< ai ^1) + a2 /i(/i2; ^2) + sh{ai) + 5/1(02). 

For the case of simplicity we consider only the case when Vi 
and V2 are finite families. 

Based on Propositions 11.2 and III.l it is not difficult to 
deduce the following simple, but general, greedy algorithm 
for constructing such an alphabet V. 

ALGORITHM: 

1) i = 0; 

■p^ = ViVJ P2; 

2) IF is empty GOTO STEP 4; 

ELSE find a set Pi € which maximises the value of 

V' B P ^ aifii{P) + a2fi2{P); 

IF maximum equals zero GOTO STEP 4; 

3) p'+i = {P\Pi : Pg V% 
i = i + l; 

GOTO STEP 2; 

4) p = {Fo,Pi,...,A-i}; 

END. 

Clearly, this algorithm can be directly adopted for more than 
two sources in X. 

Let us look how the above algorithm works in practice. 

Example in.l. Let X = [0,2]. We consider two measures 
Ml : [0, 1] ^ R and IJ,2 : jg] ^- K given by 

Hi{A) = [ Idx, ii2{A) = 2 [ (x-^) dx. (62) 
Ja Ja -LU 

As an error-control family Q we take the family of all intervals 
contained in [0, 2] with length not greater than |. We consider 
coding alphabets: 

^i = {[o,f),[|,|Ml4Mii]}> ,63, 
T> — {\-L l^ fi J-^ \J- iiu ^ 

I I L 10' 2''' 1-2' 10''' 1-10' 10''' 1-10' 10 J J" 

Mixture of sources is given by probabilities a\ =2/5 and 
ci2 = 3/5. 

The algorithm presented above produces following Q- 
acceptable alphabet of the mixture: 

T = {[0, ^), \h 5)' % D' [§' !)' [!' 1]' (1' %\}- (64) 
We get the entropies: 

ai /i(/ii ; Pi) + 0,2/1(^2; P2) « 1.93, (65) 

ax h{\iuVx) + 02 /7,(M2; V2) + s/i(ai) + 5/1(02) « 2.9, (66) 

/i(ai/ii +0,2^2;^) ~ 2.36. (67) 

As we see, we have obtained a reasonable coding method 
for finding joint alphabet of the mixture of sources. 

IV. Renyi Entropy Dimension 

From now on we always assume that X is a metric space 
and S contains all Borel subsets of X. 



A. Entropy Dimension of Convex Combination of Measures 

Entropy of a probability measure /i with respect to the 
error-control family Q G S identifies minimal amount of 
information needed to encode an arbitrary element of X with 
error Q. Renyi entropy dimension in turn gives the rate of 
convergence of this quantity when error is decreasing. Thus it 
is also important to estimate the entropy dimension of convex 
combination of measures. Making use of Theorem III.l it is 
quite simple. 

Given 5 > let us denote a family of all balls in X with 
radius 5 by 

Bs:={B{x,6):x(^X}, (68) 

where B{x, 5) is a closed ball centred at x with radius 5. 

We consider Bg as an error-control family. If we want to 
code a point x E X hy a certain ball B(q, S) then we may code 
it in fact by its centre q. Thus the error we make, simply equals 
to the distance between x and q. Consequently, the family Bs 
allows to code points from X with error not greater than S. 

For the convenience of the reader let us recall the definition 
of the entropy dimension [2]. 

Definition IV.l. The upper and lower entropy dimension of 
measure ^ e Mi {X, S) are defined by 



dims (At) := limsup — — -, 

5^-0 -l0g2(()) 

dimg(M) := liminf ^['^'^,^}. ■ 
s^o -log2(5) 



(69) 



(70) 



If the above are equal we say that jj, has the entropy dimension 
and denote it by dim e{ij). 

We apply Theorem 111.1 for estimation of Renyi entropy 
dimension of convex combination of measures. 

Theorem IV.l. Let n e N and let au £ [0, 1] for k e 

n 

{l,...,n} be such that ak = 1. If {At/c}fc=i C Mi{X, S) 
fe=i 

then 

n n 

dinig(y^ UkUk) > afcdini^(/Xfe), (71) 



fe=i 



fc=i 



dims (^ akHk) <^ Ofedim^ (/Xfe ) . (72) 
fe=i fe=i 

Proof: Let S e (0, 1) be given. By Theorem III.l, we 



have 



(73) 



fe=i 



fe=i 



and 



/ ^ dkl^k 
fe=i 



■,S) <J2akH{nk;S) + J2Mak)- (74) 



fc=i 



fe=i 



Dividing by — log2 {S) and taking respective hmits as 5 — >^ 0, 
we obtain assertion of the theorem. ■ 

Corollary IV.l. Let n e N and let E [0, 1] for 

n 

k G {l,...,n} be such that ^ = 1. Let {l^k}k=i C 

fe=i 
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Mi{X, S). If every Hk has entropy dimension for k € As dims(X) < oo, for every e > we find N gN such that 

n 

{1, . . . , n} then Yl o-kfJ-k also has entropy dimension and oo ^ 



fc=i 



dim£;(^ ttfe/ifc) = ^ afedimB(/ifc). (75) 



fe=JV+l 

Thus by Theorem IV. 1, we get: 



k=l k=l 

We generalise Theorem IV. 1 for the case of countable Jl^^ Jl^^ ak 

families of measures under an additional assumption that the dimg(/ , QfcMfc) < (/ ^Qi)dimg(^ ^ — ^'^'^^ 
upper box dimension of X is finite. It will allow us to prove 

stronger version (see Corollary IV.2) of theorem proved by oo oo 

A. Renyi [2, page 196] concerning the entropy dimension of +( ai)dim£;( =^05 iik) 

discrete measure. It is worth mentioning first the definition of i=jv+i k=N+i ^3=n+i 

upper box dimension [11]. ^ 

The upper box dimension of any non-empty bounded subset ^ — / \ , ^ — / \ , ,oii\ 

c f V A f. Ax^ < > afedimi<;(/xfe) +e < > afedimi<;(/xfe) +£. (86) 

i> of A IS defined by ^ ^ 



fe=i fe=i 



dims(5) := limsup — (76) ■ 

i->o — log Given a point a; e X, let (Jj; be an atomic measure at x, i.e. 

where Ns{S) denotes the smallest number of closed balls of f 1 if a; e A 

radius ^ that cover 5. "^^^^^ { o' if a; ^ A every A e E. (87) 

Theorem rV.2. We assume that dimnfX) < 00. Let ^, , /r \ ^ u 1 • r- 

r loo ,r / , , / r.^ n Clearly, aiuvEiOx) = for every x G A. Making use of 

{Mfe}fe=i C Mi(X, S) and let a sequence {ak)k=, C [0, 1] ^j^^^^^^ we obtain the following corollary: 

be such that ^ = 1. Then 

Corollary IV.2. We assume that dimB(X) < oc. Let 

00 00 ixk)'kLi C X and let iak)kLi ^ [Oj 1] sequence such that 

dim£;(2^afeMfe) > 2^«fedimj=;(/ifc) (77) ak = I. Then dim.E{Y. akS^,^) = 0. 

fe=l fe=l fe=l A:=l 

and ^ ^ 

-T-- — /V^ N ^ \ ^ -p — f \ /^o^ B. Improved Version of Youne Theorem 

dimB(2^afc/Xfe) < 2^afedimB(;Ufc). (78) 

fc=i fe=i Finding the Renyi entropy dimension of a given measure is 

Proof To prove first inequality we use Theorem IVl. For 1^*^^ hard task in practice. It is much easier to calculate its 
every iV e N we have: lo^^l dimension. 

The local upper dimension of /x e M\ {X, S) at point x G 

/V^ , ,,\r^ -s a-k X, is defined by 

dimB(2^afcMfc) =dim£;((2^a,)2^— p,k (79) 

£>^(a;) := hmsup . (88) 

cx) 00 logd 

1 ^* fe 1 '^'^ ^^'^ t^^J obtained an estimation of upper entropy dimension 

of Borel probabihty measure on by the supremum of local 
upper dimension, which can be seen as a version of Young 
Theorem [12]: 



AT AT 



^ . ^ ak Consequence of Young Theorem (see [10, Theorem 1.3.]) 

+{ 2^ ai)dnn£,( 2^ =^^5 TT-'^'^' For a Borel probability measure u on R^, we have 

i=N+l k=N+l ^0=N+1 3 

N N N dimB(At) < ess supD^(x). (89) 

^ (J2 XI dim^(/xfe) = E aftdim£;(/ife). 

i=i fe=i Z^j=i ^■J fe=i 

(81) 

Since N gN was arbitrary then We show that this estimation can be improved: 

^ ^ Theorem IV.3. For a Borel probability measure p, on R^, we 

dim£;(2^afe/ifc) > 2^afedim£;(/ifc). (82) f^^^^ 

fe=i fc=i 
We prove second inequahty. It is well known that if u G 
Mi{X,'E) then Proof: Let us first observe that Dn{x) is a measurable 

dimB(z^) < dimB(X). (83) function, as the mapping x — > p{B{x,5)) is measurable. 



dimB(/i)< / Di,{x)dp{x). (90) 
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Since for /x-almost all x e M^: D^{x) < N then we divide APPENDIX A 

the segment [0, A''] into n € N parts and denote sets Hardy-Polya-Littlewood Theorem 

^ _ -j^ ^ We generalise the classical Hardy-Littlewood-Polya Theo- 

:= {x : D^{x) € ( ^A^, -N]} (91) rem [13, Theorem 1.5.4.] for infinite sequences. 

Hardy-Littlewood-Polya Theorem. Let a > and let ip : 

forn G N and G {0, . . . , n - 1}. Let us define probability a] ~> R+, ^(0) = be a continuous concave function. 

Let {xi)i^i, {yi)izi C [0,a] be given sequences where either 
1 ^ c ,i(An\ ^ n / = N or 7 = {1, . . . , N} for a certain A' G N. We assume 



measures 



■ 1 , if = ^ ' 



Y^Xi <Y,yn for n & I (99) 
for n G N and i G {0, . . . , n - 1}. Since c X then i=i i=i 



and 

D^n{x)<D^{x)<^—N (93) Y.Xi = Y,yi- (100) 



for /i-almost all the points x G A^. Making use of Conse- // {xi)iei is nonincreasing sequence then 
quence of Young Theorem and (93), we have 



dim£;(/x") < ess s\xpD^,r.(x) < (94) 



Y,^{x,)>Y.^{y3)- (101) 



Proof: The classical Hardy-Littlewood-Polya Theorem 
By the definition of we represent measure as a convex Theorem 1.5.4] covers exacdy the finite sequence case, 

combination of /x^, i.e. that is when / = {1, . . . , AT} for a certain A^ G N. We will 

^ show that the case when 7 = N follows from the case when 

I is finite. 

To prove (101) it is sufficient to show that for every n G N 
there exist A;„ G N such that 

kn n 

Y,'P{xi)>Y,^{yi), (102) 

since all sequences under considerations are nonnegative. Let 
n G N be arbitrary and let A;„ > n be chosen so that 

< J2 M(^?)dim^(/.?) < ^ ^,{A:)-—^N. (97) :=Y.x.-J2y.>0. (103) 



M=^M^"K (95) 

i=0 

for each n G N. Applying Theorem IV. 1 and (94), we get 

ri-l 

m^Eii^) = dh^EiY, /"(^DmD (96) 

i=0 



n— 1 n— 1 



i=0 i=0 



i=l 



Finally taking limits as n ^ 00, we obtain Such a choice is possible since and {yi)iei are 

nonnegative sequences which have equal sum. 
fc(/x)<y D^{x)dfi{x). (98) Consider two finite sequences of equal length fc„: 

x = {xi,...,xkj and y = (j/i, . . . ,y„,r„+i, 0, . . . ,0). 

„, vc 1 1 • 1 • 1 ij Observe that the above sequences have equal sum and that 5 

We were unable to verity whether a similar estimation holds . . • , , r , ^, 

J, ^, , ^ J. . . .J, r 7-, / N 7 / N ^ IS nomncreasing. We show that for every k < kn 

for the lower entropy dimension, i.e. it IJ^(x)afj.{x) < 

dim£;(/x). ,A ,A 

^ii<^yi. (104) 

1=1 i=l 

V. Conclusion If fc < n, this follows from the assumptions made on 

. f • • , , J- sequences (xi)ig/ and {yi)iei. H k > n then 

Our paper investigates the problem ot joint lossy-coding 

of information from combined sources. The main result gives _ _ _ 

the estimation of the entropy of mixture of sources by the A^^i - a^^^ ~ z^^i ~ 2^^^ ''"+1 ~ Z^^»- (105) 

combination of their entropies. The proof is based on the new 

equivalent definition of the entropy, which allows to obtain Since {xi)i^i is a nonincreasing we can apply to sequences x, 

a convex combination of partitions contrary to the classical y and function ip the finite sequence version of the classical 

definition. We also present a practical and easy to implement Hardy-Littlewood-Polya and obtain that 

algorithm of constructing joint coding alphabet for above ^.^ ^.^ f.^ 

problem. As a corollary we generalise some results concerning ip{xi) = </5(ii) > ^(yi) (106) 

the Renyi entropy dimension. .^-^ -^^ 
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n 

= ^{Vi) + vK+i) + {kn - (n + 1))(^(0) (107) 

i=l 

n 

>Y,^{yi)- (108) 
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